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Abstract 

This study presents an application of a measure of learner involvement developed by Paas, 
Ttwvinen, van Merrienboer, and Darabi (2005). These authors combined learners’ perfor- 
mance scores with their perceived mental effort invested in instruction and used it to assess 
learner involvement in instructional conditions. The present study examines the differentiating 
attribiites of the Paas et al. learner involvement measure by using data collected in an experi- 
ment that investigated the effectiveness of three computer-based instructional strategies: (a) 
conventional problem solving, (b) product-oriented worked examples, and ( :) process-oriented 
worked examples. As hypothesized, learners using worked example strategies showed higher 
involvement scores than those in conventional problem solving. However, no differences in 
learner involvement were fotmd between the two worked-example strategies. Tlje implications 
of these findings for designing instructional strategies and suggestions for further research are 
discussed. (Keywords: involvement, engagement, motivation, instructional strategies, worked 
examples.) 

INTRODUCTION 

Theories of motivation have variously described the sources of motivation in 
the traditions of behavioral, humanistic, and cognitive psychology. Behavioral 
psychology attributes motivation to learner’s prior experiences with the environ- 
ment in response to present stimuli. The humanistic explanation considers mo- 
tivation as individuals’ fulfillment of their potentials and focuses on emotional, 
interpersonal, and intellectual self-actualization as the need for that fulfillment. 
Cognitive theories of motivation focus on peoples’ beliefs and expectations, and 
examine variables such as expectancy for success, value placed on success, goals 
and goal orientation, attribution about reasons for success, needs for compe- 
tence, control, and relatedness (Eggen & Kauchak, 2001). All three traditions 
specify that the learner’s active involvement in instruction is necessary for suc- 
cessful learning outcomes. But the question remains: How is the necessary in- 
gredient of involvement elicited and measured? 

By explaining underlying factors of learners’ behavioral, emotional, interper- 
sonal, and cognitive involvement in learning tasks, these theories have guided 
research on learners’ motivation and quality of the learning experience. In this 
line of research, the concept of “engagement” has emerged as the learner charac- 
teristic attributed to sustained, effortful, and enthusiastic participation in learn- 
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ing tasks. Even though the literature treats the concepts of learner involvement 
and engagement separately, they essentially describe the same emotional, cogni- 
tive, and behavioral characteristics. 

According to Skinner and Belmont (1993), when learners engage in learning 
tasks they demonstrate continued behavioral involvement accompanied by posi- 
tive attitude. Engaged learners select tasks that challenge their competency level, 
and they invest intense effort to accomplish those tasks. Furthermore, as Reeve, 
Jang, Carrell, Jeon, and Barch (2004) suggest, these learners are characteristical- 
ly focused, directed, goal oriented, and relentless during their interaction with 
social and environmental learning conditions. They further propose that learn- 
ers who engage in dynamic, interactive instruction are “organically” involved in 
their instructional experience. Thus we can argue that learning environments 
that employ stimulating and intriguing learning strategies are more likely to en- 
gage learners than passive instructional strategies. 

On the opposite end of the engagement spectrum are the disaffected learners 
who, as Furrer and Skinner (2003) describe, are estranged, indifferent, defiant, 
or burned out, and therefore deprived of motivation for learning. Understand- 
ing the learners’ lack of motivation due to frustrations, negative expectations, 
disappointments, and discouragements is essential in identifying the steps in 
the learning process and the conditions under which loss of motivation occurs 
(Miceli & Castelfranchi, 2000). 

The other concept to which motivation research frequently refers is learners’ 
“involvement.” For instance, in their discussion of learners’ engagement, Skin- 
ner and Belmont (1993) refer to behavioral involvement as an indicator of the 
learners’ engagement. They described learners’ engagement as “intensity and 
emotional quality of children’s involvement in initiating and carrying out learn- 
ing activities” (p. 572). The involvement concept is further characterized by 
Turner et al. (1998) as “complex interaction of student cognition, motivation 
and affect” (p. 730). Reed and Schallert (1993) also report that involved learners 
describe their learning experience as focused concentration, attention, and deep 
comprehension, to which Csikszentmihlyi, Rathunde, and Whalen (1993) add 
positive affect, goal clarity, and intrinsic motivation. 

Using the concepts of involvement and engagement interchangeably, Turner 
et al. (1998), address the issue of measuring these concepts in the literature. 
They observe that studies of learners’ cognition, motivation, and affect have fo- 
cused on cognitive engagement and involvement, even though the empirical ex- 
amination of these constructs has differed in focus. Researchers have measured 
engagement or involvement either through ratings of active involvement, such 
as effort and positive emotion, or through observations of learners’ initiatives in 
trying to take responsibility for their behavior (Reeve et al., 2004). 

Paas, Tuovinen, van Merrienboer, and Darabi (2005) described another meth- 
od of measuring learners’ involvement in the context of Cognitive Load Theory 
(CLT; see reviews by Paas, Renkl, & Sweller, 2004; Sweller, van Merrienboer, 

& Paas, 1998). The measure focused on the relative involvement of learners in 
two or more instructional conditions. It combined the learners’ mental effort 
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invested in the learning task and the cognitive load imposed by the performance 
of that task. 

The present study used Pass et al.’s (2005) methodology to examine learn- 
ers’ involvement in three instructional strategies using an existing data set. The 
strategies included two types of worked examples (process- and product-ori- 
ented worked examples) and conventional problem solving. The purpose of the 
study was to compare the relative involvement attributes of these strategies. The 
learning task was troubleshooting a computer-simulated chemical processing 
plant. We hypothesized that learners using both of the worked example strate- 
gies would have higher involvement scores than learners using conventional 
problem solving because they would be encouraged to try the new-found strat- 
egies illustrated by the worked examples rather than their own familiar strate- 
gies. We also hypothesized that the process-oriented worked example group 
would show greater involvement than the product-oriented worked example 
group because the explanation of reasons for diagnoses in the process-oriented 
worked examples would stimulate individual reasoning about problem-solving 
strategies. 

METHOD 

Participants and Experimental Setting 

Thirty-six senior engineering students enrolled in a chemical engineering de- 
sign course participated in the study as part of a required class assignment. All 
participants had taken courses that instructed concepts, rules, and principles 
of distillation. The engineering professor required his students to complete the 
practice to understand how a chemical plant operates as part of their course, but 
the activity was not graded. The professor did not specify any penalty if they 
did not complete the activity, but all students completed the activity. 

The experiment was conducted in a research laboratory specifically designed 
for research on the acquisition and transfer of complex cognitive skills using a 
computer-based simulation. The simulation provided an authentic learning en- 
vironment in which the participants played the role of the chemical processing 
plant operator (see Figure 1). As long as all sub-systems of the plant functioned 
normally, the operator’s role involved merely observing the automated opera- 
tion of the plant. However, the gradual deterioration of the plant’s components 
caused specific malfunctions that the operator diagnosed and repaired as quickly 
and as efficiently as possible. The learning environment thus afforded students 
numerous opportunities to handle several mission-critical emergencies which, 
in the actual operation of a plant, would be rare but potentially costly. 

Design and Procedure 

Participants were randomly assigned to three groups of 12. All 36 individu- 
ally received computer-based instruction on how to troubleshoot simulated 
malfunctions. This introduction consisted of self-paced, computer-based text 
and graphics to teach participants about the components, features, and trouble- 
shooting processes required to run the simulation. The time they spent in the 
self-paced instruction ranged from 21 to 56 minutes. 
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Figure 1: Mam interface of the simulation of the chemical processing plant con- 
trol panel. 


Worked examples and conventional problem solving were the contrasted 
instructional strategies. The three strategies included process-oriented worked 
examples, product-oriented worked examples (Van Gog, Paas, & van Merrien- 
boer, 2004), and conventional problem solving and were assigned to treatment 
groups Process, Product, and Problem, respectively. These three instructional 
strategies were implemented as they are described below: 

Process-oriented worked examples. The participants in the Process group were 
presented with text and graphics describing a procedure for analyzing four sim- 
ulated plant malfunctions. The instruction not only described the troubleshoot- 
ing process but also explicated principled reasoning about why steps were taken 
and how the most likely cause of the malfunction could be diagnosed. 

Product-oriented worked examples. The participants in the Product group 
were given four worked-out examples with the same malfunctions that were 
presented to the Process group. The participant’s task was to study each of the 
worked examples in a procedural step-by-step format. However, in contrast to 
the Process treatment, these worked examples stated only hoiv the problems 
were solved, but not the reasons why each step in the procedure was taken. The 
procedure included five steps for solving each problem: (a) identify the affected 
regulating loop, (b) inspect preceding and following loops, (c) identify the 
components of the affected regulating loop, (d) examine each component in 
the loop — which involved three sub-steps that concluded with identifying the 
contradictory component as the cause of the malfunction — and (e) report the 
diagnosis. 

Conventional problem solving. Rather than studying how to solve the prob- 
lem, participants in this strategy, the Problem group, started troubleshooting 
malfunctions with the fully functional simulation as their practice of skill. They 
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were asked to troubleshoot the same problems as the worked example groups 
without any description or explanation of the problems. 

Following completion of their assigned strategy, which ranged in duration 
from 8 to 46 minutes, all participants diagnosed the same set of eight novel 
malfunctions. Participants were instructed to diagnose malfunctions making as 
few incorrect diagnoses as possible. Twelve minutes were allowed for each mal- 
function. If a malfunction was not diagnosed correcdy within the allotted time, 
the simulation would end the episode and direct the participant to the next 
malfunction. The simulation software collected participants’ performance and 
mental effort data after each malfunction episode using the measures described 
below. 

MEASURES AND INSTRUMENTS 

Performance measure. For each malfunction following the instructional strat- 
egy, the software recorded whether the participants correctly diagnosed the fault 
on the first try. In the operation of an actual chemical plant, there is little room 
for operator error. The first-try-correct measure was selected as the performance 
indicator because of the operational task criticality in the actual chemical distil- 
lation environment. The performance score represented the number of first-try- 
correct responses out of a total of eight. 

Mental Effort Scale. The 9-point, single-item Mental Effort Scale (Paas & van 
Merrienboer, 1994a) measured the perceived mental effort subjects invested 
in performing the tasks. The question prompted participants to indicate the 
amount of mental effort they invested in the task they had just completed. At 
the high end of the scale, “9” was associated with the learner’s investment of 
“very, very high mental effort,” and at the low end, “1” was associated with 
“very, very low mental effort.” The scale was administered immediately follow- 
ing each malfunction, whether the malfunction was repaired or not, to provide 
a subjective rating of the participant’s “cognitive load.” Paas (1992), Paas and 
van Merrienboer (1994b), and De Crook, van Merrienboer, and Paas (1998) re- 
port Cronbach’s coefficient alpha of 0.90, 0.82, and 0.98 respectively. De Crook 
and van Merrienboer (2007) reported a coefficient of 0.88. 

Measure of learners’ involvement. The following technique was used to com- 
pute and depict learners’ involvement of instructional conditions according to 
Paas et al. (2005). Following their methodology, we converted the mental effort 
and performance scores into z scores to compute the involvement (7) measure 
according to the following equation: 

R + P 

/= "vf 

The standardized scores for mental effort ( R ) and performance (P) were rep- 
resented on a Cartesian graph with mental effort on the x-axis and performance 
on the y- axis (see Figure 2). A particular point in this coordinate system refers 
toaz score for mental effort and its respective z score for performance of a 
given instructional condition. This graph applies the above equation based on 
the computation of the perpendicular distance of a point (defined by the stan- 
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Table 1: Descriptive Statistics for Performance, Mental Effort, and 
Involvement in Eight Performance Tasks by Treatment Group 








95% Confidence 
Interval for 
Mean 

Range 



Treatment 




Std. 

Lower 

Upper 



Variable 

Group 

n 

M 

SD 

Error 

Bound 

Bound 

Min. 

Max. 

Performance 

Problem 

12 

1.08 

1.00 

0.29 

0.45 

1.72 

0 

3 


Process 

12 

1.50 

1.31 

0.38 

0.67 

2.34 

0 

4 


Product 

12 

2.25 

1.36 

0.39 

1.39 

3.11 

1 

5 


Total 

36 

1.61 

1.29 

0.22 

1.17 

2.05 

0 

5 

Mental 

effort 

Problem 

12 

5.05 

0.86 

0.25 

4.51 

5.60 

3.63 

6.25 


Process 

12 

5.54 

1.40 

0.40 

4.65 

6.43 

2.75 

7.00 


Product 

12 

5.85 

1.11 

0.32 

5.15 

6.56 

4.38 

8.00 


Total 

36 

5.48 

1.16 

0.19 

5.09 

5.88 

2.75 

8.00 

Involvement 

Problem 

12 

3.28 

0.80 

0.23 

2.77 

3.79 

2.12 

4.47 


Process 

12 

3.86 

0.82 

0.24 

3.34 

4.38 

2.51 

5.36 


Product 

12 

4.49 

1.00 

0.29 

3.86 

5.12 

2.94 

6.01 


Total 

36 

3.88 

1.00 

0.16 

3.54 

4.21 

2.12 

6.01 


dardized means of mental effort and performance for each instructional condi- 
tion) to a zero involvement line, which is graphically presented as R + P = 0. 
This graph provides a visual display of the composite measure as an indication 
of learners’ involvement, which, according to Paas et al. (2005), represents the 
motivational attributes of an instructional condition. A shift to the upper right 
of the coordinate system indicates an increase in those attributes, and a shift to 
the lower left indicates a decrease. 

RESULTS 

The data collected for the experiment described in this study were analyzed 
for the purpose of examining learners’ involvement in three instructional strate- 
gies based on Paas et al.’s (2005) technique. The descriptive statistics about the 
variables in this analysis are presented in Table 1 . 

The degree of learner involvement was calculated by the aforementioned 
equation using the standardized scores of first-try-correct performance and of 
mental effort. Analysis of variance was used to examine the learners’ involve- 
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Mental Effort 


Figure 2: Relative involvement properties for instructional conditions: process- 
oriented worked examples (PC), product-oriented worked examples (PD), and 
conventional problem solving (PB). The diagonal line represents the central 
composite measure of involvement. 

ment in the three different strategies. The omnibus test showed a significant dif- 
ference among the strategies, F (2, 23) = 5.95, p = .006. A planned contrast was 
analyzed to explore the difference between the conventional problem-solving 
group (Problem) and the worked example groups (Process and Product). As- 
suming equal variance, the planned contrast indicated that the learners’ involve- 
ment in the worked example conditions were significantly different from those 
in the conventional problem condition (t = -2.88, df= 33, p = .007). To inves- 
tigate the possibility of a difference between the two worked example groups 
(Process and Product), the two strategies were contrasted against one another 
while holding the Problem strategy constant. Again assuming equal variances, 
the difference between the two conditions was not significant with alpha set at 
the .05 level (t = -1.91, df= 33, p = .065). 

To illustrate the relative involvement of the learners in these strategies we then 
used the involvement component variables, performance and mental effort, 
to plot a graph as described in the Method section. The standardized scores of 
first-try-correct as the performance measure and the invested mental effort for 
each group were plotted on the x- and j/-axes, respectively (see Figure 2). 
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DISCUSSION 

The results supported our first hypothesis that learners using worked example 
instructional strategies would show greater involvement than the learners using 
the conventional problem solving strategy. The measure of involvement was 
based on the Pass et al. (2005) metric, which was developed in the context of 
CLT, and used learners’ invested mental effort and their performance scores as 
components of the metric. The involving features of worked examples found 
in this study complemented the positive instructional contributions of worked 
examples found in the literature (Atkinson, Derry, Renkl, & Wortham, 2000; 
Sweller, Van Merrienboer, & Paas, 1998). 

Motivation literature lists attributes of involvement or engagement as (a) sus- 
tained, effortful, and enthusiastic participation, (b) positive attitude, (c) intense 
effort, (d) focused attention, and (e) goal directedness (Reeve et al., 2004; Skin- 
ner & Belmont, 1993). Learners who demonstrate any or all of these attributes 
might be motivated by either performance or learning goals. Accordingly, the 
composite of standardized mental effort and performance scores may be con- 
sidered as a measure of learner involvement for the comparison of instructional 
strategies or other variables. 

With regard to cognitive load theory, the underlying involvement measure 
used in this study relies on the supposition that investment of mental effort 
in learning tasks, commensurate with the appropriate level of germane cogni- 
tive load (cognitive demand of instruction that contributes to learning), results 
in better performance of learning tasks. Extraneous cognitive load (cognitive 
demand of instruction that inhibits learning), on the other hand, results in 
lower performance scores. Given these principles and the measures use of both 
mental effort and learners’ performance scores, we suggest that the higher level 
of involvement indicated by worked examples represents the germane, or effec- 
tive, cognitive load imposed by these strategies, hence their greater instructional 
value from behavioral, cognitive, and emotional standpoints. The increased 
involvement of learners using worked examples may have derived from the 
focused, goal-directed approach to problem solving that the worked examples 
modeled. 

Our second hypothesis predicted that participants using process-oriented 
worked examples would demonstrate higher involvement than participants us- 
ing product-oriented worked examples. Contrary to our expectation, there was 
no significant difference between the two strategies. We suspect that the similar 
involving features of the two worked example strategies or the small sample size 
was responsible for this lack of significant differences. A valid pretest of par- 
ticipants’ prior knowledge about troubleshooting the distillation process might 
have provided a covariate for examining with analysis of covariance the strate- 
gies’ contributions to subjects’ performance and involvement. 

Overall, this study provided an example of how the composite measure of in- 
volvement recommended by Paas et al. (2005) can differentiate between learn- 
ers participating in different instructional conditions. Furthermore, considering 
learners involvement as a motivational attribute of the instructional strategy, 
the findings indicate that worked examples might be more motivating than 
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the conventional problem solving strategy. For better support of this point, we 
suggest that this measure of learners’ involvement be contrasted with other well- 
established measures of instructional motivation. An example would be Keller’s 
(1987) Instructional Materials Motivation Survey (IMMS) that analyzes learner 
perception of the motivational aspects of instructional materials in terms of at- 
tention, relevance, confidence, and satisfaction. 

Pending such research, this measure is one of the useful tools for differentiat- 
ing between instructional strategies or groups of learners in terms of their in- 
volvement. This measure may be helpful to instructional designers in their anal- 
ysis of learner characteristics and formative evaluation of instructional materials. 
In pilot testing of any instructional module, one can gain considerable informa- 
tion about the appropriateness of the content to learner characteristics as well as 
the level of involvement of the learners among alternative instructional methods 
or materials by using this approach. 
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